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REMARKS 

Claims 1-13 arc pending in the application. Claims 1-13 are rejected. All 
rejections and objections are respectfully traversed. 

The Examiner rejected claims 1-13 under the judicially created doctrine of double 
patenting over daim 1 of U.S. Patent 6,546,135, "Method for representing and 
comparing multimedia content" issued to Lin et al., on April 8, 2003, in view of 
Lee, et al., (Querying multimedia presentations based on content, IEEE Trans, on 
Knowledge and Data Engineering, vol. 1 1, no. 3, May/June 1999. 

First, the Examiner will note that the present application is a Continuation-in-Part of 
U.S. Patent 6,546,135. 

Second, Lee is irrelevant to what is claimed. As explained below in the Applicants' 
traversal of the Examiner's 35 U.S.C. 103(a) rejection based on Lee, Lee never 
measures attributes of content entities. At page 362, Lee describes "querying 
presentation graphs using temporal operators." The invention assigns measured 
attributes to each corresponding content entity in the directed acyclic graphs to rank 
order the multimedia content. Lee only queries the graph. The invention produces a 
graph having rank ordered content entities. Lee only queries graphs. Lee never 
describes any steps to produce a directed acyclic graph having rank ordered content 
entities as claimed. The Examiner's reference to specifying paths using 
computational tree logic is a querying step and does not produce a directed acyclic 
graph, and is therefore irrelevant to the invention. 
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Further, the Examiner erroneously points to an order of appearance of video streams 
in a presentation as describing rank ordering. A temporal ordering has nothing to do 
with the rank ordering content entities based on measured attributes as claimed. Lee 
takes as input a graph with video streams having some temporal ordering. Ordering 
video streams temporally says nothing about the relative rank of the videos. 
Nowhere does Lee describe that the temporal order in which the videos appear in the 
presentation has anything to do with the relative rank of the videos. 

Lee never changes the order of the video streams as they appear in the presentation, 
nor does Lee assign a rank order to the video streams. Lee never rank orders 
anything. 

The invention orders multimedia content. Image or video multimedia content is 
segmented to extract objects. Features of the objects are then extracted and 
associated to produce content entities. The content entities are coded to produce 
directed acyclic graphs of the content entities, each directed acyclic graph 
representing a particular interpretation of the multimedia content. Attributes of 
each content entity are measured and the measured attributes are assigned to each 
corresponding content entity in the directed acyclic graphs to rank order the 
multimedia content. 

Furthermore, there is no motivation for combining Lee with Lin, by Lee's own 
admission: 

In this paper, we assume that multimedia presentations 
are created and stored in the form of multim^ia presenta- 
tion graphs, which can be viewed as high level abstractions 
for multimedia presentations. Simply, a multimedia pres- 
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Lee is already handed his graph, Lee doesn't need Lin. Therefore, a terminal 
disclaimer is not required. 

Claims 1-13 are rejected under 35 U.S.C. 103(a) as being unpatentable over Lee, et 
al., (Querying multimedia presentations based on content, IEEE Trans, on 
Knowledge and Data Engineering, vol. 1 1, no. 3, May/June 1999 - "Lee"). 

It should be noted that Lee was a primary reference, in combination with Miller, in 
the first office action dated July 17, 2003. The rejection was successfiilly traversed 
by the applicants and the Examiner withdrew the rejection and issued a new 
rejection based on completely different (and inapplicable) art. That rejection was 
successfully traversed, and now the Examiner has produced Lee again, with no 
other support, in yet another 35 U.S.C. 103(a) rejection. 

MPEP 707.07 explicitly states "Piecemeal examination should be avoided as much 
as possible." Further, MPEP 904.03 states "In all references considered, including 
nonpatent, foreign patents, and domestic patents, the examiner should study the 
specification or description sufficiently to determine the fiill value of the reference 
disclosure relative to the claimed or claimable subject matter." 

It appears that the Examiner is attempting to inappropriately "stretch" the Lee 
reference, which is resulting in excessive prosecution efforts by the applicants. 

Lee is now applied by the Examiner with no other supporting reference, but 
coupled with a statement that "it would have been obvious to rank order" in view 
of Lee. MPEP 7070.07(g) states "Where a major technical rejection is proper, it 
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should be stated with a full development of reasons rather than by a mere 
conclusion coupled with some stereotyped expression." The rejection by the 
Examiner is a mere conclusion, without a fiill development of reasons. The 
Examiner has repeatedly foiled to produce prior art teaching the invention, and the 
applicants respectfully request the Examiner to allow the application instead of 
continuing to unnecessarily prolong the prosecution based on multiple 
interpretations of inapplicable references in an attempt to "fit" the reference to the 
claim 

The Examiner should understand at this point, as stated in the response to the first 
office action, that Lee can only begin after the invention is finished. The invention 
generates graphs of multimedia content. Lee queries already existing graphs of 
multimedia presentations for content. Lee is totally silent on how and where the 
graphs were created and stored. 

Lee describes an object-oriented programming language for querying content of 

graphical presentations; see page 361, right column, lines 17-22 - page 362, left 

column, lines 1-3, below: 

In this papei; we assume that multimedia presentations 
are oeated and stored in the form of multimedia piesenta- 
tion graphs, whidi can be viewed as high level abstractions 
for multimedia presentations. Simply, a multimedia pres- 
entation graph specifies the playout order of various types 
of streams making up the multimedia presentation, i.e., it is 

a visual specification of a presentation plan. Using a graph 
model for presentations, this paper discusses languages for 
tfuerytng multimedia presentation graphs. 

Lee never produces graphs. Lee takes graphs as input. A person of ordinary skill in 
the art would know that querying a graph can never describe segmenting 
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multimedia content, extracting objects, extracting features of objects, associating 
the extracted features of the objects to produce content entities, coding content 
entities to produce directed acycHc graphs of the content entities, measuring 
attributes of each content entity, or assigning the measured attributes to each 
corresponding content entity in the directed acycHc graphs to rank order the 
multimedia content. The multimedia querying described by Lee is useless for 
making the invention obvious. 



The input to Lee is a graph. Lee can never describe segmenting the multimedia 
content to extract objects as claimed. Lee just queries the graph. Lee never 
segments anything. Figure 1 of Lee shows a presentation graph, nothing more. The 
presentation graph shows an arrangement of video and audio streams that miake up' 
a presentation. Lee never describes segmenting multimedia content or extracting 
objects as the Examiner asserts. 
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The Examiner has mischaracterized the labels of the boxes in Figure 1 of Lee. A 
person of ordinary skill in the art would never confuse an object extracted from 
multimedia content as claimed, with a manually assigned title of a video stream. 
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Even Lee refers to the video streams by title, see below: 

streams. Fig. 1 gives an example of a simple ptosiaitaSim 
graph entitled "National Geography " consisting of video 

^-?TJ'^*^f ^^^'^ Yellowstone", "YeUowstone", 
WUdUfe^, "Landscapes", "Forests", and "Next week in 
National Geography", and audio streams 'Tromo Song" 
and "Four Seasons". In this paper, we Ulustratc our contri- 
butions using only the video multimedia data type. 

There is no description, suggestion, or even a hint that the titles in Figure 1 are 
somehow extracted from the content of the videos. 



The only way that Lee could extract the title would be to study the video content, 
such as these exemplary Yellowstone seasonal scenes, and extract the weather 
features: 



Winter Spring Summer Fall 




and detect the winter snows, and then watch the winter change to spring, summer 
an fall, as the color of the leaves change, and then analyze a scene of the park 
entrance. 




and then Lee's 'multimedia database system' says, aha, from my in-depth analysis 
of scenes in the content of this video, it is obvious to this very clever multimedia 
database system that the content clearly indicates that the title of this video must 
be: 

"Four seasons in Yellowstone" 
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Applicants are skeptical that this is the case, and that Lee's multimedia database 
system has this capability. The invention does. 

The Examiner's depiction of video stream titles as extracted video objects is clearly 
erroneous. Therefore, the rejection should be reconsidered and withdrawn, because 
Lee never segments multimedia content, or extracts objects as claimed. 

Claimed is extracting and associating features of the objects to produce content 

entities. Therefore, each content entity is made up of different associated features 

of each object. For feature extraction, the Examiner points to Lee at page 362, left 

column, second full paragraph, below: 

A multimedia video stream consists of a 'sequence of 
video frames, each of which is assodafed with some amtent 
mfomwtioiv namdy, a set of content objects and cortent te^ 
tetjonships among its content objects. So, our object-oriented 
data model includes presentation graph, stream, frame and 
contentK>bject classes whose objects represent; respectively, 
multimedia presentafion graphs, multimedia streams, video 

frames and content objects. Presentation node is also a dass 
and inherits attributes of the stream class. 

There is no description of extracting or associating features, above. Lee describes 

an object oriented data model that includes classes for graphs, video streams, 

frames and content objects. Lee's data model is used for querying content and has 

nothing to do with feature extraction or association. 

The Examiner is requested to specifically point out exactly which words mean 
extracting and associating features of the objects to produce content entities, as 
claimed. , 

The invention codes the content entities to produce directed acyclic graphs (DAG) 
of the content entities, each directed acyclic graph representing a particular 
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interpretation of the multimedia content. Lee never produces DAGs. Lee only 
queries DAGs that were assumedly handed to him. Lee never generates DAGs 
representing interpretations of multimedia content based on content entities. Lee 
produces query results. Lee is useless for making the invention obvious. 

As stated above, Lee takes as input a graphical representation of a multimedia 
presentation. There is no description of generating graphs anywhere in Lee. Lee is 
silent on content entities according to the invention, segmentation, and measured 
attributes associated with corresponding edges. According to Lee "Each 
presentation stream is a node in the presentation graph and edges describe 
sequential or concurrent playout of streams." See page 363, right col., 3"'/4* 
paragraphs. The Examiner is respectfully requested to explain exactly which words 
at page 363, right col., 3'%* paragraphs, mean coding content entities to produce 
directed acyclic graphs of the content entities. This is impossible for Lee, because 
Lee never extracts objects from multimedia content, and Lee never extracts and 
associates features from extracted objects to produce the content entities that are 
coded as claimed. Lee is irrelevant to what is claimed. 

The invention measures attributes of each content entity and assigns the measured 
attributes to each corresponding content entity in the directed acycUc graphs to 
rank order the multimedia content. So the output of the invention is directed 
acyclic graphs including rank ordered content entities. The Examiner's repeated 
assertion that the text in Lee stating "a multimedia presentation graph specifies the 
playout order of various types of streams making up the multimedia presentation" 
suggests rank ordering the multimedia is nonsensical. Playout order is simply a 
temporal order by which each video stream is played so that the presentation 
makes sense, e.g., part one is played before part two. Temporal playout order of 
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video streams never suggests attribute based rank order of content entities in a 
DAG. The rejection makes no sense and should be reconsidered and withdrawn. 



Regarding claim 2, Figure 2 of Lee does not include the words intensity or order, 
increasing or decreasing, anywhere. Applicants respectfully request the Examiner 
explain how Fig. 2 even suggests intensity attributes or rank ordered attributes. 
Applicants cannot find any words that suggest intensity or rank ordered attributes 
in Figure 2. 



name: String; 

Bdgettj {<1^«c»JCodo} ] ; 
class Stream H^pe ( 



fnanoK <Fnxai^>; 
iKU>fJramfi6: Ist^gcr; 
topic String; 
date: Date; 

elaaa Pres^ode : inherits from Stream 
other cUfihut^ ); 



dftaa FVam« type ( 
name: String, 
objecU! {C-Object}; 
0ther attri^e^]; 

ctea* CObjcct typ^ { 
name: String; 
type: String; 

oiher aitHbutcs |; 



Notatum: "{ f denote the get constntctor, and «< denotes the sequence constroctor. 



no. Z Data motfaftiQ or a iKrattaSoA 



Regarding claim 3, the two special attributes described by Lee sre Nodes and 
Edges. This has nothing to do with object direction and there is no suggestion to 
measure or assign the attributes. 



Regarding claims 4 and 5, the same is true for temporal and spatial attributes. Lee 
compares queries to attributes. The Examiner is requested to specifically point to 
where Lee measures temporal or spatial attributes, assigns attributes, or rank orders 
content entities based on attributes. The Examiner point to temporal operators, 
which are a tool in object oriented programming applications. Operators have 
nothing to do with attributes. 
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In claims 6 and 7, the measured attributes are arranged in an increasing (claim 6) 
or decreasing (claim 7) rank order. The Examiner points again to Lee describing a 
playout order of a multimedia presentation. Lee has nothing to do with rank 
ordering as claimed. The invention traverses the multimedia content and 
summarizes the multimedia content according to the directed acyclic graph and the 
measured attributes assigned to the content entities. As stated above, Lee never 
measures or assigns attributes. Lee uses temporal operators, such as ''Next, 
Connected. UntiF to query a presentation. Lee begins where the invention ends. 

In claim 8, the multimedia content is traversed according to the directed acyclic 
graph and the measured attributes assigned to the content entities. Lee searches 
graphs based on queries. Lee never describes assigned attributes. Lee never 
assigns anything to a content entity. 

In claim 9, the multimedia content is summarized according to the directed acyclic 
graph and the measured attributes assigned to the content entities. Lee only 
queries multimedia presentations. Lee never summarizes multimedia content. The 
Examiner's reference to graph components at page 363 is irrelevant. 

In claim 10, the multimedia content is a three dimensional video sequence. The 
Examiner's rejection of claim 10 is a clear indication that the Examiner is simply 
recycling the first office action. The rejection points to a section of Lee that does 
not exist. Rather, Applicants believe that the section is erroneously lifted fi-om 
Miller, in the first office action, a reference already withdrawn by the Examiner. 

In claim 1 1, nodes of the directed acyclic graphs represent the content entities and 
edges represent breaks in the segmentation, and the measured attributes are 
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associated with the corresponding edges. Lee never describes segmenting 
multimedia content. Further, Lee's edges represent the end of video streams. As 
claimed, edges rq)resent breaks in the segmentation. 

In claim 12, at least one secondary content entity is associated with a particular 
content entity, and wherein the secondary content entity is selected during the 
traversing. The Examiner's rejection of claim 12 is yet another indication that the 
Examiner is simply recycling her first office action. The rejection again points to a 
section of Lee that does not exist. Rather, this section is also erroneously lifted 
from the now defimct Miller. 

In claim 13, a sunnmary of the multimedia is a selected permutation of the content 
entities according to the associated ranks. The Examiner's rejection of claim 13 is 
still another indication that the Examiner is simply recycling her first office action. 
The rejection again points to a section of Lee that does not exist. Rather, this 
section is also erroneously lifted from the now defunct Miller. 

In view of the foregoing, it is respectftilly submitted that the application is in 
condition for allowance and an early indication of the same is courteously 
solicited. The Examiner is respectfiiUy requested to contact the undersigned by 
telephone at the below listed telephone number, in order to expedite resolution of 
any remaining issues and fiirther to expedite passage of the application to issue, if 
any fiuther comments, questions or suggestions arise in connection with the 
application. 

To the extent necessary, a petition for an extension of time under 37 C.F.R. § 1.136 
is hereby made. Please charge any shortage in fees due in connection with the 
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filing of this paper, including extension of time fees, to Deposit Account 50-0749 
and please credit any excess fees to such deposit account. 



Respectfully Submitted, 
Mitsubishi Electric 
Research L^p^tojies, Inc. 




Andrew J. Curtin 
Registration No. 48,485 



201 Broadway, 8* Floor 
Cambridge, MA 02139 
Telephone: (617) 621-7573 
Facsimile: (617) 621-7550 
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